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(54) A non-timeline, non-linear digital multimedia composition method and system 



(57) A non-temporal, non-linear multimedia compo- 
sition method and software system used in producing 
digital composition sequences. The method treats ail 
digital signals and information including video, audio., 
image, documents source materials, as common data 
types for composition and creation purposes, The inven- 
tion employs a spatial, non-temporal, non-linear story 
line method for displaying and manipulating digital 
source material objects for composition purposes. The 
method utilizes content specific, dynamic pictorial ob- 
jects to denote digital source materials stored in com- 
puter readable format to facilitate ease of operation in 
performing non-linear, non-timeline multimedia compo- 
sition operations. Methods are provided for dynamic 
previewing and manipulation of images, video, audio, 
three dimensional objects and animation sequences de- 
noted by dynamic pictorial objects representing digital 



source materials stored in computer readable format. 
The method of the present invention employs a function- 
specific software console metaphor in which all compo- 
sition elements are displayed within a single, integrated 
console environment, including a dynamic story line dis- 
play facility, video editor, special effects editor, transi- 
tions editor, video animation editor, audio studio, plug- 
in editor, digital capture and output editor. The present 
invention includes a method for visually denoting the ac- 
tive status of one or more composition editors as applied 
to a specific composition segment or sequence. Meth- 
ods for facilitating dynamic image capture from digital 
video sources are employed in the present invention. 
The system and method utilizes animated multistage 
three dimensional pictorial icons with associated prop- 
erties as controls for the purpose of initiating specific 
computing commands and composition process related 
tasks. 
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Description 

COPYRIGHT NOTICE 

A portion of the disclosure of this patent document 
contains materia! which is subject to copyright protec- 
tion. The copyright owner has no objection to the repro- 
duction by anyone of the patent document or the patent 
disclosure as it appears in the Patent and Trademark 
Office patent files or records, but othenvise reserves all 
copyright rights whatsoever. 

FIELD OF THE INVENTION 

The present invention relates generally to the field 
of information processing and, more particularly to a 
method and system for multimedia composition. 

BACKGROUND OF THE INVENTION 

In the past decade : video and audio magnetic tape 
have emerged as viable alternatives to traditional stor- 
age media such as silver halide film for recording, dis- 
play and reproduction of images, sounds and video. Ed- 
iting methods in the past originally stemmed from linear 
or sequential editing of film on a film device. Along with 
advances in storage media have come new methods of 
composition based on computerized digital editing tech- 
niques that have all but replaced manual editing of mag- 
netic video and audio tape and related media. Digital 
composition software programs, such as those used to 
compose digital video, audio and related source mate- 
rials, have found widespread application in multimedia 
presentation and authoring systems. Despite their pow- 
er, these composition software programs require users 
to master complex interfaces, and adopt abstract repre- 
sentation models which are beyond the grasp of ordi- 
nary users. 

For example, some prior art composition software 
programs include user interfaces that employ timeline- 
based, non-linear and non-spatial representations of 
what is inherently a temporal-based composition proc- 
ess making the composition software programs difficult 
to learn and use. The user interfaces are non-linear be- 
cause various aspects of the composition are displayed 
as separate elements on different levels or layers. 
These composition software programs seek to mimic 
film based editing methods through the application of a 
timeline-based composition system and method. 

However, unlike the film editor who has access to 
the visual scenes of a movie which he can cut and splice 
together to produce a desired sequence : the typical vid- 
eo editor has been forced to rely on computer software 
programs that simulate sequential frames of film on a 
timeline. The conventional non-linear digital editing 
method involves the use of computer software for dis- 
playing and manipulating a sequence of frames or im- 
ages on a timeline typically with each frame individually 



displayed. Where multiple sequences are sought to be 
confined, each sequence is displayed on a separate, 
horizontal timeline. Audio to accompany the video is 
placed on a separate timeline. The video and/or audio 
5 sequences contained in each timeline are not viewable 
until a production stage is completed. While this method 
offers significant productivity gains over the complex 
and time-consuming task of manual editing, the timeline 
based method utilized in current non-linear digital video 

10 and audio composition systems is extremely complex, 
non-intuitive and cumbersome to use because of the 
multiple timelines or layers. 

One of the problems with the prevailing non -linear 
composition method is that it breaks a composition into 

is a non-temporal representation of multiple layers of im- 
age and/or audio sequences in what is inherently a tem- 
poral relationship. This method not only forces users to 
constantly switch between separate program modules 
to perform routine composition tasks for each layer, but 

20 greatly increases the complexity of the composition dis- 
play due to the inherent need to accommodate multiple 
timelines or "layers", each with its own associated im- 
ages. A user is forced to mentally visualize the com- 
bined layers as well as their effects as editing is per- 

25 formed. Thus the composition process is, as a conse- 
quence, very complex and tedious. As a result, much of 
the power of these composition software programs is 
untapped by ail but the most advanced users due to the 
inherent complexity that results from attempting to rep- 

30 resent a spatial and temporal composition process by 
means of non-spatial, timeline-based models and meth- 
ods. 

U. S. Patent No. 4,533,188 to Barker et. al. disclos- 
es a video composition method and apparatus utilizing 
35 non-linear timeline based composition. Each layer of the 
video composition method is presented as numerous in- 
dividual images in a time sequence or timeline order. 
There are many inherent difficulties and inefficiencies 
associated with the prior art method that results from the 
40 display of temporal source materials in a non-temporal 
or timeline fashion. 

However, despite its inherent limitations, the use of 
digital non-linear editing methods has increased signif- 
icantly in recent years due to associated cost advantag- 
es es offered over traditional video editing methods. Unfor- 
tunately, this departure from traditional video editing 
techniques has in some respects dampened the crea- 
tive talents of modern video and audio directors who are 
no longer able to apply their skills directly to the compo- 
50 sition process and must instead rely upon intermediate 
personnel to manipulate the image and audio sequenc- 
es scattered across multiple timeline layers. Using the 
non-linear method of digital-based composition, the di- 
rector's feel for the composition process is greatly dimin- 
55 ished. One advantage of composing with the traditional 
physical editing method using silver halide film, is the 
ability to react quickly to the temporal nature of the me- 
dia. Thus, a given sequence of film can be run back and 
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forth, viewed repeatedly, edited and quickly "spliced" to- 
gether with other sequences to form a completed seg- 
ment. Thus the source material is treated as an entire 
object, one that can be combined with other objects to 
create a composition sequence. The ability to achieve s 
this same effect in a digital non-linear editing environ- 
ment is highly desirable. 

It is therefore an object of the present invention to 
provide a novel non-timeline, non-linear digital multime- 
dia method and system. 10 

SUMMARY OF THE INVENTION 

According to one aspect of the present invention 
there is provided a method for non-timeline, non-linear is 
digital multimedia composition comprising the steps of: 
representing multimedia materials stored in memory as 
objects; retrieving one or more of the objects from the 
memory for display on a composition console; dynami- 
cally viewing and editing at least one or more objects on 20 
the composition console to produce edited object; and 
placing the edited objects on a storyline area of the com- 
position console to create a storyline, the storyline being 
viewable on the composition console. 

In another aspect of the present invention there is 25 
provided a system for non-timeline, non-linear digital 
multimedia composition, the system comprising: means 
for representing multimedia materials stored in memory 
as objects; means for retrieving one or more of the ob- 
jects from the memory for display on a composition con- 30 
sole; means for dynamically viewing and editing one or 
more objects on the composition console to produce ed- 
ited objects; and means for placing the edited objects 
on a storyline area of the composition console to create 
a storyline, the storyline being viewable on the compo- 35 
sition console. 

In still yet another aspect of the present invention 
there is provided a computer readable media encoded 
with a computer program for creating non-timeline, non- 
linear digital multimedia compositions from multimedia 40 
materials stored in memory as objects comprising: 
means for retrieving one or more of the objects from the 
memory for display on a composition console; means 
for dynamically viewing and editing the one or more ob- 
jects on the composition console to produce edited ob- 45 
jects; and means for placing the edited objects on a sto- 
ryline area of the composition console to create a sto- 
ryline, the storyline being viewable on the composition 
console. 

In yet another aspect of the present invention there 50 
is provided a method for producing a multimedia com- 
position comprising the steps of: a) providing a user in- 
terface having a source material object window and a 
storyline strip, the storyline strip having a plurality of po- 
sitions; b) associating individual identifiers in the window 55 
with time varying dynamic multimedia source material 
objects; c) choosing one of the identifiers in the window 
to retrieve a selected object; d) dragging the chosen 



identifier across the interface to one of the positions on 
the storyline strip; e) repeating steps (c) and (d) to or- 
ganize a multi-object, multimedia composition in a de- 
sired order; f) generating an output composition from se- 
lected objects organized on the storyline strip. 

In still yet another aspect of the present invention 
there is provided a system for creating a multimedia 
composition from multimedia source material objects 
comprising: a user interface having: a tool bar for ac- 
cessing system functions; a source library window for 
accessing multimedia source material objects; a con- 
sole for previewing and editing the multimedia source 
material objects to create output objects; and a storyline 
for assembling the output objects to form an output com- 
position; a processor for editing said source material ob- 
jects and for creating and organizing the output compo- 
sition in response to user input entered via said user 
interface; a display for displaying the output composition 
on the console. 

In still yet another aspect of the present invention 
there is provided a system for producing a multimedia 
composition comprising the steps of: means for gener- 
ating a user interface having a source material object 
window and a storyline strip, the storyline strip having a 
plurality of positions; means for associating individual 
identifiers in the window with time varying dynamic mul- 
timedia source material objects: means for choosing 
one of the identifiers in the window to retrieve a selected 
object: means for placing the chosen identifier on one 
of the positions on the storyline strip; means for organ- 
izing a multi-object, multimedia composition in a desired 
order: and means for generating an output composition 
from selected objects organized on the storyline strip. 

In still yet another aspect of the present invention 
there is provided a computer readable media encoded 
with a computer program for producing a multimedia 
composition from multimedia materials stored in mem- 
ory as source material objects comprising: means for 
generating a user interface having a source material ob- 
ject window and a storyline strip, said storyline strip hav- 
ing a plurality of positions; means for associating indi- 
vidual identifiers in said window with time varying dy- 
namic multimedia source material objects; means for 
choosing one of said identifiers in said window to re- 
trieve a selected source method object from memory; 
means for placement of one or more selected said iden- 
tifiers to selected positions on said storyline strip to or- 
ganize a multi-object, multimedia composition in a de- 
sired order; and means for generating an output com- 
position from selected objects organized on said sto- 
ryline strip. 

The non-timeline, non-linear digital composition 
system and method allow digital source material to be 
produced while providing the operator with instantane- 
ous feedback. This significantly increases productivity, 
while at the same time lessens or removes the need for 
intermediate personnel. This also provides a highly in- 
tuitive and creative digital composition environment for 
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non-technical users, editors and directors alike. The 
present composition method and system also provide a 
dynamic, intuitive and user-friendly object based com- 
position environment for producing digital source mate- 
rials for use with a wide variety of commercial applica- 5 
tions, media and transmission mediums. 

The present composition system includes system 
software control functions that are highly user interac- 
tive and respond to user commands for selecting audio, 
video and multimedia objects from source materials, 10 
displaying these dynamically in a main viewing area, 
performing one or more composition tasks or steps, ap- 
plying such composition tasks dynamically to the object, 
previewing the composite object and placing the fin- 
ished video, audio and related multimedia source ma- is 
terial sequences in a production storyline facility for real- 
time preview of the sequence composition, and for pro- 
duction of the completed sequence. 

The composition system and method of the present 
invention treat digital source segments, including still 20 
frame images, video, audio, three dimensional objects, 
animation and related multimedia source materials as 
"digital objects" and allow for both temporal and accu- 
rate display. Furthermore, the composition system and 
method of the present invention allow for application of 25 
special effects "filters" that can be applied equally and 
dynamically to all digital objects in the composition proc- 
ess. Audio special effects filters can be applied directly 
to an audio file which in turn can be applied to an ani- 
mated three dimensional video object such that it caus- 30 
es the object to react or modify its motion in a manner 
consistent with an audio (i.e. music) source file. 

The present composition system and method do not 
rely on representing sequences as multiple static picto- 
rial frames. Instead, the present composition system 35 
and method directly parallels the techniques employed 
in traditional physical film-based composition where ed- 
itors view and work directly with temporal video and au- 
dio "objects" or film sequences that can be played back 
and forth, viewed, edited and enhanced in a dynamic, 40 
real time composing environment. Digital objects can be 
placed on the storyline in any desired order and can be 
dynamically manipulated and previewed by the operator 
at any time. Each digital object can be associated with 
a pictorial identifier. The identifier for the object can be 45 
drawn from any desired pictorial image source. In the 
case of video segments, the identifier representing a 
given object can be a single video frame selected either 
automatically by the system or manually by the operator. 
The present invention does not use separate frames or 50 
pictorial labels to denote the start and endpoint of given 
source material segments or sequences in a timeline as 
is characteristic of conventional, timeline-based digital 
composition systems and methods. 

In the case of audio sources, representative object ss 
images are selected by the user from a library of repre- 
sentative audio images or automatically generated 
based on an analysis of the segment's audio wave pat- 



terns. Source materials and compositions referenced in 
the system's source material storage libraries can be 
grouped and sorted according to user specified criteria. 
* Users can search for source materials stored on both 
local storage media as well as on-line and network me- 
dia storage devices according to wide range of user de- 
fined criteria and object properties. 

BRIEF DESCRIPTION OF THE DRAWINGS 

An embodiment of the present invention will now 
described more fully with reference to the accompany- 
ing drawings in which: 

Figure 1 A is a block diagram of computer system 
for executing a non -timeline, non -linear digital mul- 
timedia composition program in accordance with 
the present invention; 

Figure 1 B is a block diagram illustrating the relation- 
ship between the computer system of Figure 1 A and 
a computer software system; 
Figure 1 C is a bitmap screen shot of prior art com- 
position interface illustrating a typical timeline- 
based, non-linear digital composition program; 
Figure 1D is a bitmap screen shot of primary com- 
position user interface of the non-timeline, non-lin- 
ear digital multimedia composition program execut- 
ed by the computer system of Figure 1A including 
three dimensional tool bar icons, a source library 
component, a composition storyline component and 
a primary viewing console: 

Figure 1 E is a bitmap screen shot of the animated 
three dimensional tool bar icons of Figure 1 D which 
can be activated to invoke functions; 
Figure IF is a bitmap screen shot illustrating an ex- 
ample of the implementation of animation of one of 
the three dimensional tool bar icons of Figure 1 D; 
Figure 2A is a bitmap screen shot of the source li- 
brary component of Figure 1 D used to store source 
materials and productions; 

Figure 2B is a bitmap screen shot of the composi- 
tion storyline component of Figure 1D illustrating 
how finished sequences and transition effects are 
arranged to create a finished composition se- 
quence; 

Figure 2C is a bitmap screen shot of the primary 
viewing console of Figure 1 D used to display source 
materials and compositions and to manipulate vid- 
eo and audio segments using various composition 
tools accessible from the primary viewing console; 
Figure 2D is a bitmap screen shot of primary view- 
ing console controls of Figure 2C used to manipu- 
late source materials (i.e. play, pause, advance, 
etc.) as well as time code indicators to set in and 
out points used in the editing of source materials for 
composition purposes; 

Figure 2E is a bitmap screen shot of primary viewing 
console controls of Figure 2C used to access vari- 
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ous composition tools and to display the active sta- 
tus of such tools; 

Figure 2F is a bitmap screen shot of list of object 
properties accessible from the primary composition 
interface of Figure 1 D; 

Figure 3A is a bitmap screen shot of video editor 
controls of the primary viewing console of Figure 1 D 
illustrating how color control variations are per- 
formed along with related composition operations; 
Figure 3B is a bitmap screen shot of transition com- 
position controls of the primary viewing console of 
Figure 1 D; 

Figure 3C is a bitmap screen shot of text editing 
controls of the primary viewing console of Figure 
1D; 

Figure 3D is a bitmap screen shot of transition ed- 
iting primary viewing controls of the console of Fig- 
ure 1 D; 

Figure 3E is a bitmap screen shot of animation con- 
trols of the primary viewing console of Figure 1 D; 
Figure 3F is a bitmap screen shot of audio compo- 
sition controls of the primary viewing console of Fig- 
ure 1 D; 

Figure 3H is a bitmap screen shot of video capture 
controls of the primary viewing console of Figure 
1D; 

Figure 4A is a bitmap screen shot of composition 
pre-production controls on the primary viewing con- 
sole of Figure 1D; 

Figure 4B is a bitmap screen shot of the composi- 
tion production controls of the primary viewing con- 
sole of Figure 1D; 

Figure 5A is a bitmap screen shot of general system 
options accessible from the primary composition in- 
terface of Figure 1D; 

Figure 5B is a bitmap screen shot of source material 
dynamic preview options and controls accessible 
from the primary composition user interface of Fig- 
ure 1D; 

Figure 5C is a bitmap screen shot of composition 
production options and controls accessible from the 
primary composition user interface of Figure 1 D; 
Figure 5E is a bitmap screen shot of source material 
capture or acquisition options and controls acces- 
sible from the primary composition user interface of 
Figure 1D; and 

Figure 5F is a bitmap screen shot of content capture 
driver options and controls accessible from the pri- 
mary composition user interface of Figure 1 D. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

Referring now to Figure 1A, a computer system 10 
for executing a non-timeline, non-linear digital multime- 
dia composition program in accordance with the present 
invention to produce multimedia compositions is shown. 
The composition program when executed allows a user 



to select video, audio and multimedia objects from 
source materials and display the selected objects dy- 
namically. Composition tasks can be performed and ap- 
plied to the objects to create composite objects which 
5 can be previewed. Selected and composite objects can 
be placed in a production storyline for real-time preview 
Specifics of the composition program and the computer 
system 10 will now be described. 

As can be seen in Figure 1 A, computer system 10 

10 includes a central processor 1 1 , a main memory 1 2, an 

~" input/output controller 1 3, a keyboard 1 4, a pointing de- 
vice 15 such as mouse, track ball, pen device, voice ac- 
tivated command system or the like, a display device 
1 6, a mass storage disk memory 1 7 such as a hard disk, 

is a printing device 18 and, a video and audio input and 
output subsystem 21 for connection of devices such as 
audio and video cassette recorders, cameras, camcord- 
ers, microphones and the like. In the preferred embod- 
iment, the pointing device 15 is a two-button mouse de- 

20 vice, including both right and left 'mouse buttons. Pro- 
gramming techniques and operations for mouse devic- 
es are well documented in the programming and hard- 
ware literature; such as Microsoft Mouse Programmer's 
Reference, Microsoft Press, 1989. Additional input/out- 

25 put devices, such as a digital tape (not shown), devices 
for access to live video and audio streams from network, 
internet and broadcast media; and related systems may 
be included in the computer system 10 as desired. 
As illustrated, the various components of the com- 

30 puter system 1 0 communicate through a system bus 20 
or similar architecture. In the preferred embodiment, the 
computer system 10 includes an IBM(g>-compatible per- 
sonal computer, which is available from multiple ven- 
dors (including IBM of Armonk, N. Y). It is within the 

35 scope of this invention that other computer designs can 
be used. 

In response to user movement signals from the 
mouse 1 5, a pointer floats (i.e. moves freely) across the 
display device 16 to a desired location. During or after 

40 cursor movement, the user may generate user-event 
signals (e.g. mouse button "clicks" and "drags") for se- 
lecting and manipulating on-screen objects, elements 
and controls, in a manner as is well known in the art. 
As shown in Figure 1 B, computer software system 

45 30 is provided fo>directing the operation of the computer 
system 10. Computer software system 30, is stored in 
main memory 1 2 and on mass storage disk memory 1 7, 
and includes a kernal or operating system 31 , and a 
shell or interface 33. Application software 32, in this 

50 case the composition program, may be "loaded" (i.e., 
transferred from disk memory 17 into system memory 
12) for execution by the computer system 10. The com- 
puter system 10 operates in accordance with instruc- 
tions from operating system 31 and/or the composition 

55 program. The interface 33 : which, in the preferred em- 
bodiment: is a graphical user interface (GUI) : is used to 
display results. It is also through interface 33 that the 
user may supply additional inputs, commands or termi- 
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nate the computer session. In the preferred embodi- 
ment, operating system 31 and interface 33 consists of 
Windows 95™ and related programs; which are availa- 
ble from Microsoft Corporation of Redmond, Washing- 
ton. 

For ease of understanding, a prior art timeline, non- 
linear digital composition program will firstly be de- 
scribed with reference to Figure 1 C. As can be seen, the 
composition program has a user interface 100 including 
a system menu 104, a program menu 106 and a timeline 
area 102 containing multiple timelines. The system 
menu 104 provides open, close, save and general op- 
erators for the whole application. The program menu 
1 06 launches function specific dialog windows using a 
specific applet interface transferred to each function that 
when activated, appears and resides on top of the time- 
line area 102. 

Un it time bar 1 08 provides the user with control over 
the time increments in the sequentially organized imag- 
es present in the timeline area 102. As the user increas- 
es the time, the timeline collapses showing fewer imag- 
es in the timeline area 102. A time ruler 110 displays the 
time scale of the current production. Video layers 112 
and 114 of timeline area 102 are two examples of the 
multiple layers where the user places and manipulates 
existing video segments of the production. Each video 
segment consists of multiple sequential frames. Each 
frame of the production appears separately and horizon- 
tally across the timeline area 1 02. FX layer 1 1 6 is where 
the user deposits an iconic representation of transition 
effect such as a dissolve, wipe, etc. This effect is gen- 
erated using a specific transition effects dialog interface 
from program menu 106 and is then applied to the FX 
layer 116. Sliders 118 operate both vertically and hori- 
zontally and are used for building multiple special effects 
by adding layers to the timeline area 102. When many 
effects are added, information which is not in view on 
the screen must be accessed using sliders 118 to scroll 
the window to see the hidden information. The horizon- 
tal slider allows the user to view the sequential images 
on the timeline out of view of the screen. 

Video layers 120 and 122 are two examples of plu- 
rality of layers containing a repository of special effects 
icons which are applied to video layers 112 and 114. It 
is with the video layers 1 20 and 1 22 that various special 
effects are applied, after they have been generated us- 
ing function specific dialog windows from program menu 
106. 

Audio layers 124 and 126 are where the audio por- 
tion associated with the composition are represented. 
They are also where additional audio files may be added 
as separate layers to be applied to the video layers, 112 
and 114. 

Special effects icons 128 and 130 represent a par- 
ticular effect that has been introduced to effect the video 
residing in video layers 112 and 114 and FX layer 116. 
Each special effect icon 1 28 and 1 30 is associated with 
a certain range of times on time ruler 110. 



Shortcut buttons 132 allow for manipulation of the 
time base for the composition, and to other functions 
available on system menu 104 used to edit the produc- 
tion. 

5 As can be seen, the prior art non-linear digital com- 
position program employs a series of discrete, special 
purpose composition editors (not shown) accessible 
through system menu 1 04 that are utilized independent- 
ly of the timeline presentation display to perform basic 

10 composition tasks such as applying text and video ani- 
mation effects to a given composition. Each composition 
task or step requires a separate timeline sequence or 
■layer' such as video layers 112. 114, 120 and 122, FX 
layer 116 and audio layers 124 and 126 to be created 

'5 and displayed for each sequence element needed that 
is used to produce a final composite video or audio se- 
quence. The separate audio and video layers are not 
combined at editing time, so that the combined result is 
not known until the playback stage of the process. As 

20 will be appreciated, this prior art method involves selec- 
tively operating upon the segments on at least two sep- 
arate layers or levels, each with its own associated time- 
line. Furthermore, this method requires grouping of se- 
rially connected segments, appearing in layers 1 1 2, 1 1 4, 

25 116, 120, 122, 124 and 126 into groups that have the 
same timeline relationship. These multiple layers are 
difficult to manage. 

The non-timeline, non-linear multimedia composi- 
tion program executed by computer system 10 over- 
do comes the disadvantages associated with the above- 
described prior art composition program and will now be 
described with reference to Figures 1 D to 5F. When the 
present compsoition program is executed, a graphical 
primary composition user interface 40 is displayed on 

35 the display device 16 as illustrated in Figure 1 D. 

The user interface 1 40 includes four major areas, 
namely: a tool bar 150; a source library window 180; a 
console 200 having a viewing window 205; and a sto- 
ryline 170. This structure simplifies the receiving, 

40 processing and presenting of information, including dig- 
ital video, audio and related composition data and pro- 
vides straightforward controls for manipulating and out- 
putting a multimedia composition. A pointer 185 for se- 
lecting and otherwise invoking screen objects of interest 

45 is also provided and is responsive to the mouse 1 5. User 
interface 140 may be closed, minimized or system user 
help facilities invoked by activating screen close button 
166, minimize button 167, or help button 168 respec- 
tively. The tool bar 150 features multi-stage three dimen- 

50 sional animated icons which can be activated to provide 
quick access to common system functions such as 
opening and saving compositions, generating produc- 
tions and manipulating objects. The source library win- 
dow 180 displays and organizes one or more specific 

55 digital source objects which can be selected, displayed 
and manipulated on the console 200. The console 200 
is used for previewing and editing specific digital source 
objects including adjusting images, applying special ef- 
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fects, adding and adjusting fonts, blending objects, ma- 
nipulating audio and capturing video, images and audio 
from outside source. The storyline 170 is used to as- 
semble and manipulate digital source objects, edited 
objects and transitions into a composition for produc- 
tion. 

To create a composition, a user uses the tool bar 
1 50 to access system functions to create a new compo- 
sition or edit an existing one. The source library window 
1 80 is used to access various multimedia source mate- 
rials and place them as objects on the console 200. Ed- 
iting tools on console 200 are used to edit and apply 
special effects to the objects. The console 200 is also 
used to create and manipulate transitions between ob- 
jects. Objects and transitions displayed on the console 
200 can be moved to the storyline 1 70 and manipulated 
to form a composition. The tool bar 150 is used to save 
the composition and generate a final output production. 

In response to operator commands through compu- 
ter system 1 0, source library window 1 80 allows the user 
to "scroll" through lists of source materials displayed as 
pictorial labels, alphanumeric file names or both repre- 
senting actual source material objects accessible from 
computer system 10, mass storage device 17 or input/ 
output subsystem 21. Using "drag" and "drop" object 
manipulation techniques, the desired source material 
can be placed in primary viewing window 205 for dy- 
namic preview purposes and for performing a multitude 
of composition-related editing tasks, as well as posi- 
tioned in the storyline 170. 

Further specifics of the user interface 140 will now 
be described. 

TOOL BAR 

Turning to Figure 1 E, the tool bar 1 50 is better illus- 
trated. As can be seen, the tool bar 150 includes a col- 
umn of icons 151 to 159 respectively. The icons can be 
activated to invoke a number of functions as will now be 
described. Icon 151 is activated to commence a new 
production, providing a blank storyline 170 into which 
objects, transitions and special effects can be applied. 
Icon 152 is activated to open a previously saved or re- 
corded composition. Icon 153 is activated to save the 
currently active composition on the storyline 170 to 
memory of system 10. Icon 154 is activated to convert 
the currently active composition on the storyline 1 70 into 
a single finished production in a multimedia format such 
as AVI : MPG, MOV etc. and output the finished produc- 
tion to memory of system 10 or input/output system 21 
for later playback. Icon 1 55 is activated to cut a selected 
object such as a video clip, transition etc. from the sto- 
ryline 170 to the clipboard of software system 30. Icon 
1 56 is activated to copy a selected object such as an 
audio or video clip, transition etc. from the storyline 170 
to the clipboard. Icon 157 is activated to paste a selected 
object such as an audio or video clip, transition etc. from 
the clipboard to the currently active storyline 170. Acti- 



vation of icon 158 deletes the currently selected object. 
Icon 159 activates the undo function, undoing the last 
user interface operation. 

Turning to Figure 1F, the three dimensional aspect 
s of the icons 151 to 1 59 of toolbar 1 50 is illustrated. The 
icons 151 to 159 activate both software system 30 and 
user controllable properties. These include sound, mo- 
tion, process monitoring, context-sensitive display, 
background tasks, etc. Icons 151 to 159 can be pro- 

io grammed in such a way as to respond to the changing 
information presentation needs of the user. 

Each of the icons 151 to 159 of tool bar 150 has 
three states. The first state is that of a standard two di- 
mensional icon. Upon moving the pointer 185 via point- 
's ing device 15 in close proximity, each of icons 151 to 
1 59 gives the illusion of moving outward forming a three 
dimensional representation. Upon activation, each of 
icons 151 to 159 rotates and returns to its original posi- 
tion. The illusion of outward motion and rotation is per- 

20 formed by the temporal display of a sequence of images 
153a in relief whereby the icon is shown with a shadow 
behind in various states of rotation. A sequence of im- 
ages 153a illustrating the above illusion of outward mo- 
tion and rotation is shown in Figure 1 F shows the typical 

25 sequence for icon 1 53. This three dimensional relief and 
rotation provides two benefits. First, it indicates to the 
user that the selection of specific icon has been recog- 
nized by the system. Second, the rotation provides feed- 
back to the user that not only has the icon been selected, 

30 but that the operation associated with that icon has been 
recognized and activated by the system. 

The tool bar 150 also includes a start button 160 
(see Figure 1D) which provides access to a pull down 
menu (not shown) relating to standard software system 

35 interface functions. Each of icons 151 to 159 provides 
a shortcut to invoke functions which are also accessible 
through the start button 1 6b. Upon activation of start but- 
ton 160, a start menu is displayed. The start menu has 
menu options for production, file library, tools, help, op- 

40 tions and print to video. Submenus relating to production 
are accessible by activation of the production menu op- 
tion from the start menu. From the production submenu : 
there is accessible an additional sub-menu containing 
a file, edit and insert clip commands relating to produc- 

45 tion. From the file menu option, standard file commands 
are available. Also, when the file menu option is select- 
ed, a produce menu option appears which, when acti- 
vated, generates a production of the currently active 
composition on storyline 170. 

50 When the library menu option is selected, com- 
mands relating to the source library window 180 are ac- 
cessed. Such commands relate to the library area 180 
and include file, edit, view, add files and insert into pro- 
duction. From the file sub-menu of the library menu op- 

55 tion, the operations of creating a new library, opening 
an existing library and saving the current active library 
displayed in source library window 180 are performed. 
From the edit sub-menu of the library menu option, the 
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features of the currently active library item in source li- 
brary window 180 are accessed. The edit sub-menu 
from the library item of the start menu includes a prop- 
erties option that provides for displaying of features of 
the currently active object. 

SOURCE LIBRARY WINDOW 

Turning to Figure 2A, source materials are shown 
as objects within source library window 180 and can be 
organized into one or more object groupings or libraries. 
Source materials may be organized according to a mul- 
titude of user defined properties such as subject, object 
type, composition sequence, file type, file name, size, 
date, etc. The source library window 1 80 is context sen- 
sitive in that it displays source materials which are of the 
same type as the currently active object on the console 
200. For example, if a video object is active in the con- 
sole, video source material objects are presented in the 
source library window 1 80. If a transition appears on the 
console, various transition effects which can be applied 
appear in the source library window 180. If an audio ob- 
ject is presented on the console, audio effects which can 
be applied to the object appear in source library window 
180. 

Source objects may be organized using hierarchical 
and relational database file storage methods for rapid 
retrieval and content management purposes. By posi- 
tioning the pointer 1 85 in the described position and right 
"clicking" with the mouse 15, users can access the soft- 
ware system functions to perform a multitude of tasks. 
Such tasks include opening, creating and saving new 
libraries, viewing source files either as dynamic images 
or alphanumeric file names, adding files to any given 
library, causing selected files to be placed in the viewing 
window 205 of console 200 for previewing purposes, 
causing selected source materials to be automatically 
positioned on the storyline 1 70, enabling users to per- 
form cut, copy, paste and delete operations as applied 
to selected source materials, renaming source material 
files and displaying given scene properties according to 
associated file and media properties. Many correspond- 
ing operations such as open file, open library, cut, copy 
and paste are also available using start button 160 of 
tool bar 150. 

STORYLINE 

Turning to Figure 2B, the storyline 170 is better il- 
lustrated and as can be seen, a "film strip" 171 metaphor 
is utilized. Client windows are located on the storyline 
1 70 and represent viewing frame objects 172 constitut- 
ing a source material and segment sequence. The client 
windows are separated by transition frames 173 on the 
film strip 171. The transition frames 1 73 are used to de- 
note the presence of transition sequences between two 
sequential viewing frame objects 172 as well as to pro- 
vide access to an associated transition composition ed- 



itor tool by means of double mouse click. Source mate- 
rial segments can be dynamically rearranged and ma- 
nipulated by means of "drag" and "drop" techniques on 
the film strip 171 allowing a user to quickly arrange com- 
s position segments into a desired sequence. The film 
strip 171 sequence display can be manually advanced 
in any direction by positioning the pointer 185 and se- 
lecting and dragging the "film strip 0 171 in the desired 
horizontal direction. A user can also advance to the be- 
10 ginning or end of the composition sequence by means 
of "Home" and "End" shortcut keyboard commands. In- 
dividual viewing frame objects 172 or transition frames 
173 can be highlighted for dynamic preview in the pri- 
mary viewing window 205 by performing a 'shift 0 key- 
is board command in conjunction with appropriate cursor 
placement. Highlighting any given viewing frame object 
172 or transition frame 173 and performing a left and 
then a right mouse click operation will cause an associ- 
ated properties client window to appear. The client win- 
20 dow is used to perform related tasks such as dynamic 
previewing, renaming source filenames, editing transi- 
tions and composition production. 



25 



CONSOLE 



Turning to Figure 2C, the console 200 is better il- 
lustrated. The console 200, in addition to the viewing 
window 205, includes a timing control area 252, an ed- 
iting tool area 210 having a plurality of editing tool icons 

30 and a preview control area 220. Primary composition 
editing tools are accessed by means of single mouse 
click on the appropriate editing tool icons. The compo- 
sition editing tools when "activated" by the user appear 
as seamless extensions of the console 200. This con- 

35 sole adjustment is commonly known as "morphing" . In 
this manner, the console 200 includes the controls for 
all the editing needs in one place, in a consistent user 
interface, without the need for cascading menus. 

Turning to Figure 2D, preview control area 220 and 

40 timing control area 252 are better illustrated. Preview 
control area 220 includes controls that are self-illuminat- 
ing "buttons" designed to provide the user with visual 
feedback as to the current active or inactive status of 
each control or series of controls. These controls include 

45 a record button 224, a pause button 222, a play button 
221, a previous frame button 226, a next frame button 
227, a single frame advance button 228, a first frame 
button 225, and a loop button 223. Buttons 221 to 228 
are designed to mimic standard VCR controls to operate 

50 on the object active in viewing window 205. These con- 
trols are activated by a single mouse click and an allow 
an object to be dynamically displayed and played on 
console 200. 

In time control area 252, video and audio object se- 
55 quences are assigned internal time codes using the 
clock within computer system 10 for composition editing 
purposes unless the source material object already in- 
corporates previously assigned industry SMPTE time 
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code location signals. If SMPTE time code signals have 
been assigned, they are automatically displayed in con- 
junction with the source object. Slider window 231 pro- 
vides the user with a visual segment display progress 
indicator and control for manually advancing or forward- 
ing of segments displayed within the viewing window 
205 by means of cursor and mouse control. Control win- 
dow 232 incorporates a numeric display and button for 
indicating a selected start point of the object. The setting 
of a start of a given object is done in a non-destructive 
manner in conjunction with video editor 211. Control 
window 233 incorporates a numeric display and button 
for indicating a stop point and includes an editing control 
for determining the end point of an object in a non-de- 
structive manner in conjunction with video editor 211. 
Time control area 252 also contains a time code display 
window 230, progress indicator ball and slider window 
231 , segment start or "in" point indicator and control win- 
dow 232, and segment end or "out" point window indi- 
cator and control window 233. 

Turning to Figure 2E, the editing tool icons of editing 
tool area 210 are better illustrated. As can be seen, the 
editing tool icons represent a series of integrated com- 
position editing tools, including a video editor 211 , a spe- 
cial effects editor 21 2, a text editor 21 3, a transitions ed- 
itor 214, a video animation editor 215, a audio editor 
216, and a video capture editor 217. The editing tool 
icons include an accompanying "active" status indicator 
which appears as a light emitting diode to provide the 
user with visual confirmation that the editing tools have 
been used to alter a source material object displayed in 
the viewing window 205. Edit information is stored with 
each object and is used when the object is dynamically 
displayed or played on console 200 and when the com- 
position is finally produced. 

Turning to Figure 3 A, the features of video editor 
21 1 are shown. The video editor includes controls for 
adjusting aspects ofthe source material object displayed 
in the viewing window 205 by manipulating the bright- 
ness of brightness slider 310, red control slider 311, 
green control slider 31 2 and blue control slider 31 3. The 
sliders 31 0, 31 1 , 31 2 and 313 include a sliding ball which 
the user manipulates by means of pointer and mouse 
controls to adjust the corresponding color intensities of 
the source object. Additional tools are available by ac- 
tivating trim button 31 5 for "trimming" a given source ob- 
ject by denoting its start position with the button in con- 
trol window 232 and stop position with the button in con- 
trol window 233. The source segment may be split by 
split button 314 in the same manner as "trimming". This 
preserves the original object being viewed in viewing 
window 205 while simultaneously causing an identical 
copy of the object to be placed on the next sequentially 
available position of storyline 170. Duration window 316 
is also provided for setting and denoting the duration of 
the object segment currently active within the viewing 
window 205 . By activating apply button 31 7, the desired 
editing operation can be applied to the object. The orig- 



inal object settings can be reset or removed by activat- 
ing remove/reset button 318. The editing tool can be 
closed by activating close button 31 9. Buttons 31 7, 318 
and 319 apply to ail of the editing tools in editing tool 
s area 210 and operate in a like manner. 

Turning to Figure 3B, the features of special effects 
editor 212 are shown. The special effects editor 212 is 
integrated within the console 200 and allows special ef- 
fects filters' or processes to be applied to a given 
10 source object selected from a library in source library 
window 180 of Figure 2A. The special effects filters are 
represented by pictorial images for use in conjunction 
with any given object currently being displayed within 
the viewing window 205. The duration and intensity of 

*s the selected special effect at the start of display of the 
object and at the end of display is determined by means 
of start slider 321 and end slider 322. Sliders 321 and 
322 utilize a sliding ball and duration scale and the effect 
created by the special effects editor 212 is dynamically 

20 previewed or demonstrated utilizing the viewing window 
205. This provides the user with immediate feedback 
and visual confirmation of the special effect. Once de- 
termined, the special effect sequence can be applied by 
activating apply button 317. 

25 Turning to Figure 3C the features of text editor 21 3 
are shown. The text editor 213 is integrated within the 
console 200 and allows both static and animated text 
within any given source object or composition. Text win- 
dow 330 permits the user to enter text to be incorporated 

30 to source video objects displayed within the viewing win- 
dow 205. As text is entered, it is dynamically displayed 
in viewing window 205 to provide immediate user feed- 
back. Control button 330a enables the user to select text 
from a system list of font types and sizes. Font buttons 

3S 330b and 330c permit the user to designate whether font 
effects are to be applied to individual lines of text or 
groups of text respectively. 

Text overlays are dynamically positioned for appli- 
cation to video as illustrated in position control pad 331 . 

40 This enables the user to dynamically position text and 
determine both duration properties and motion control 
paths of the text across viewing window 205 as part of 
an animation sequence. Start button 332 in conjunction 
with the position of movable pointer on control pad 331 

45 determines the "start" position and properties such as 
transparency, color, etc., of the initial application of text 
to the active source object in viewing window 205. Hold 
button 333 relates to the period between the start and 
finish of the display of the object through its sequence. 

so The finish button 334 relates to the end of the display of 
the active object through its sequence. Hold button 333 
and finish button 334 operate in a similar manner to start 
button 332 and are used to control the "hold" and "finish" 
properties and positions of the text as the object is dis- 

55 played in viewing window 205. Transparency slider 335 
enables the user to vary text transparency levels, while 
text color box 337 provides access to the system color 
palette for application to text and font data previously 
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determined by the user. Duration box 336 allows the us- 
er to control the duration of each text animation "event- 
as applies to position control pad 331 , start button 332, 
hold button 333 and finish button 334 respectively. 

Turning to Figure 3D, the features of transitions ed- s 
itor 214 are shown. The transitions editor 214 is inte- 
grated within the console 200. The transitions editor en- 
ables the user to create a transition 'effect' between two 
source object segments for composition purposes. 
Transition "effects" are displayed as pictorial images 10 
and are selected by the user for application from the cor- 
responding "transitions- library accessed through and 
displayed in source library window 180. When selected, 
the image indicator of the chosen transition appears in 
client window 342 as a looping or automatically repeat- is 
ing animated sequence above two film strips. The initial 
source object is placed by the user in viewing window 
205 with a corresponding pictorial image indicator or 
representative "frame" subsequ entry appearing in view 
window 340. The second source object segment is then 20 
placed in viewing window 205 with a subsequent image 
indicator or frame appearing in view window 343. The 
duration of the transition effect can be determined and 
adjusted either by manipulation of a representative "film 
strip" in client window 342 denoting the degree of over- 25 
lap occurring between the two object segments, or by 
entering a numeric value in client window 342. Once 
completed, the effect can be "applied" to the sequence 
and subsequently viewed by the user in viewing window 
205 by activating the apply button 317. The sequence 30 
can then be placed on the storyline 1 70 for composition. 

Turning to Figure 3E, the features of video anima- 
tion editor 215 are shown. The video animation editor 
215 is integrated within the console 200. The purpose 
of the video animation editor is to enable one or more 35 
"foreground" source animation objects to be applied to 
a background source animation object. To determine the 
background of the video animation sequence, the user 
selects a given source segment from source library win- 
dow 1 80 and causes the object to be placed in viewing 40 
window 205. The corresponding representation ap- 
pears in client background window 350. The source ob- 
ject to be overlaid or animated is then selected in the 
same manner and correspondingly appears in viewing 
window 205, its relative size being determined using a *s 
"slider" ball in slider window 354. A representation of this 
source segment appears in a client foreground window 
351 . The user can remove or blend colors in the back- 
ground and foreground to achieve the desired animation 
effect using image blend button 352 and remove color so 
button 353 respectively. This "chroma keying" technique 
is well known to those skilled in the art. The animation 
path, duration and properties of the source segments or 
sequences are determined by means of position control 
pad 331 , start button 332 : hold button 333, finish button 55 
334, duration box 336 and object transparency slider 
335 which operate in a similar manner as described pre- 
viously with respect to text editor 21 3. Once completed, 
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the effect can be "applied" to the sequence by activating 
apply button 317 and subsequently viewed by the user 
in viewing window 205. The sequence can then be 
placed on the storyline 170 for composition purposes. 
Turning to Figure 3F, the features of audio editor 

216 are shown. The audio editor 216 is integrated with 
the console 200. The audio editor 216 enables users to 
add one or more audio 'tracks' to a given source object 
or composition sequence. Activation of audio editor 21 6 
causes a context-sensitive source library listing audio 
objects to be displayed within source library window 
1 80. The desired audio source object is selected by the 
user and placed in a list window 360 for composition dis- 
play purposes. Any audio source object segments or 
files already associated with the video source object 
segment being displayed in viewing window 205 are list- 
ed within list window 360. One or more audio source 
object segments can be dynamically added to the video 
object currently displayed within viewing window 205. 
Audio objects can be manipulated by selectively activat- 
ing muting box 362, fade-in box 363, fade-out box 364 
and repeat box 365. The amplitude of each audio object 
and/or track" of the object can be manipulated by the 
user by means of volume control slider 366. Corre- 
sponding "level" indicators for both left and right stereo 
volume levels are displayed on left hand indicator 367I 
and right level indicator 367r to provide the user with a 
visual representation of audio sound levels by "chan- 
nel". 

Once completed, the desired audio source object 
segment can be "applied" to the video sequence by ac- 
tivating apply button 31 7 and viewed and listened to by 
the user through console 200. The object in viewing win- 
dow 205 can then be placed on the storyline 170 for 
composition purposes. Checkbox 360a allows the user 
to choose which track of multi-track audio track dis- 
played in list window 360" is to be manipulated with re- 
spect to the object in viewing window 205. An audio am- 
plitude meter 361 displays the amplitude of the audio 
when played. 

Turning to Figure 3H, the features of capture editor 

21 7 are shown. The capture editor 217 is integrated with 
the console 200. The capture editor enables users to 
readily acquire or "capture" video and audio source seg- 
ments from external sources by means of video and au- 
dio recording devices connected to input/output system 
21 . The computer editor 217 enables the user to switch 
or toggle between video and audio capture capabilities 
by means of audio button 370. The user can dynamically 
capture an image from any video source segment dis- 
played in viewing window 205 by activating image but- 
ton 371 . Sequences of varying duration can be captured 
by activating sequence button 372. The system can op- 
tionally control external devices such as camcorders, 
VCRs and "MCI" devices connected to input/output sys- 
tem 21 by activating MCI capture button 373a in con- 
junction with selecting the appropriate device connected 
to the computer system 10 from a list of device drivers 
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displayed in driver list box 373b. Once captured, audio 
and video objects can be added to content source librar- 
ies as objects through source library window 180 and 
actual composition "productions - on storyline 170 by 
means of activating checkboxes 376 and 377 respec- 
tively. Video source button 374, when activated brings 
up a display box to choose the source device, VCR, 
camcorder, etc., connected to video input/output system 
21 to perform the capture. Compression button 375 al- 
lows for the selection and display of a list of video com- 
pression techniques to be applied to the captured video. 

Turning to Figure 2F, the features of an object prop- 
erty list 260 are better illustrated. The object property 
list 260 is accessed by first selecting an object from the 
source library window 180 or film strip 171 , then activat- 
ing the properties option on the user interface 140 by 
means of a right click of pointing device 15 or from the 
edit submenu accessible from start button 160. The ob- 
ject properties list 260, displays file properties 262, clip 
properties 264, and media properties 266 relating to the 
currently selected object or library item. 

COMPOSITION PRODUCTION 

Turning to Figure 4A, a dialog box for manipulation 
of the composition production output is shown. After 
completing the composition editing process described 
above, the user selects the production menu item from 
the start button 160 on tool bar 1 50. The user is prompt- 
ed for an identifier for the output and the desired video 
format. Once that is entered, the dialog box of Figure 
4A is presented enabling the setting of composition pro- 
duction parameters. The user can select the frame size 
from frame size list box 410 to adjust pixel width and 
height for the production. Color control for the number 
of colors is provided by selecting the desired color but- 
ton in color selection area 411. Clip scaling options are 
controlled by selecting the desired clip scaling button in 
clip scaling selection area 412. The video compression 
technique such as uncompressed AVI, MPEG, etc. is 
chosen by selecting the desired method from list box 
41 3. Data rate box 41 7 allows for the data transfer rate, 
in bps, to be set for the production. The rate of display 
of frames can be controlled and optimized by choosing 
the desired rate from frame rate box 414. Quick com- 
press box 4 1 5 allows for the selection of quick compres- 
sion to be applied to the production. The key frame box 
416 allows for the choice of the frequency for reference 
or "key" frames to be generated in the production. Pro- 
duction quality can be adjusted by means of variable 
slider 418. Audio format menu list box 41 9 allows for the 
selection of the selected audio output format for the pro- 
duction. Access to advanced encoder options supplied 
by a third party such as Microsoft is accomplished by 
clicking on an Advanced Options button 420. Pressing 
a "next 0 button 421 takes the user to the next step in the 
composition production process : as illustrated in Figure 
4B. 



Turning to Figure 4B, the screen for controlling com- 
position production is shown. The current output file 
name or device, which can be modified by the user, ap- 
pears in file name display window 401 . The numeric file 
£ length value is displayed in box 402. The source seg- 
ment or "clip" is displayed in clip start box 403. The cur- 
rent frame rate is displayed in frames box 404. The total 
number of frames in the given composition sequence is 
shown in frame box 405. The estimated file size is 
10 shown in file size box 406. The user can proceed to the 
next step in the composition production process by plac- 
ing the pointer on the "Produce" button 408 and clicking 
the mouse 15. The final composition is then generated 
and stored in mass storage device 17 or input/output 
subsystem 21 . 

Turning to Figure 5 A, the screen for displaying and 
controlling various user defined options is shown. These 
are accessed by activating the start button 160 on tool 
bar 150. The user defined, options which can be manip- 
ulated fall into numerous categories as discussed be- 
low. General options can be accessed by activating gen- 
eral options tab 501 , source material preview options 
can be accessed by activating preview options tab 502. 
Composition production options can be accessed by ac- 
tivating produce tab 503. Capture options can be ac- 
cessed by activating capture options tab 504. Finally, 
capture driver options can be accessed by activating 
capture driver tab 505. System preference management 
options are accessible from tab 501 and include: display 
tool tips : show splash screen on startup, clear library 
contents on new productions, and save library with pro- 
duction option settings which are invoked by means of 
activating the appropriate checkbox in preferred check- 
box area 506. 

Turning to Figure 5B ; the preview options accessi- 
ble upon activation of preview options tab 502 are 
shown. Preview options illustrated are default source 
video segment presentation size in pixels selected from 
size buttons box 510. Clip scaling options are set by se- 
lecting the appropriate button in clip scaling area 511. 
The video editor preview display window 512 employs 
a slider and representative speed scale to enable the 
user to adjust the frame rate to optimize viewing speed. 
The storyline preview options 513 can be adjusted by 
selecting the desired checkbox options to show effects, 
show image blends, show text/video animation. Video 
display frame rates can be adjusted with a slider and 
representative speed scale in slider window 514. 

Turning to Figure 5C, the default produce options 
accessible upon activation of produce options tab 503 
are shown. The user can select the frame size from 
frame size list box 540 to adjust pixel width and height 
for the production. Color control for the number of colors 
is provided by selecting the desired color button in color 
selection area 542. Clip scaling options are controlled 
by selecting the desired clip scaling button in clip scaling 
selection area 544. The video compression technique 
such as uncompressed AVI : MPEG, etc. is chosen by 
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selecting the desired method from list box 546. Data rate 
box 548 allows for the data transfer rate, in bps, to be 
set for the production. The rate of display of frames can 
be controlled and optimized by choosing the desired 
rate from frame rate box 550. Quick compress box 552 
allows for the selection of quick compression to be ap- 
plied to the production. The key frame box 554 allows 
for the choice of the frequency for reference or "key" 
frames to be generated in the production. Production 
quality can be adjusted by means of a variable slider 
556. Audio format menu list box 558 allows for the se- 
lection of the selected audio output format for the pro- 
duction. Access to advanced encoder options supplied 
by a third party such as Microsoft is accomplished by 
clicking on the Advanced Options button 560. 

Turning to Figure 5E, the capture options accessi- 
ble upon activation of capture options tab 504 are 
shown. The default capture directory can be set in di- 
rectory window box 520. The preview video frame rates 
can be adjusted by selecting the appropriate box in 
checkbox area 521 . The general options area 522 allow 
the user to "open catalog at startup" and "stop capture 
preview when window not active" by allowing the user 
to check the associated checkbox. 

Turning to Figure 5F, the capture driver system op- 
tions accessible upon activation of capture driver tab 
505 are shown. The capture driver system options illus- 
trated in Figure 5F include check boxes for assigning 
video capture preferences in video control area 530, in- 
cluding source material frame rate, capture limits in 
terms of length of time, capture to memory, capture with 
audio, prompt before starting, and preview options re- 
spectively. The capture file can be set in browse window 
531 enabling the user to pre-allocate storage location 
and file size. Display list window 532 enables the user 
to select the audio format of captured audio source ma- 
terials, while video source button 533 to select the de- 
sired video source from input/output subsystem 21 or 
mass storage 17. Video compression button 534 allows 
the user to select a variety of industry standard com- 
pression options by means of "drop down" menu list. 

The digital video, audio and multimedia composi- 
tion program described herein provides the user with re- 
markable flexibility and ease of use in the manipulation 
and composition of digital source materials. The assem- 
bled material can be gathered from numerous sources, 
including on-line storage devices, analog to digital sig- 
nal capture devices, scanners, compact discs and dig- 
ital cameras, then rapidly edjted and assembled in any 
desired order to create complete compositions. The us- 
er can reorder scenes, trim, shorten or lengthen se- 
quences, add new source materials, transitions, anima- 
tion's, titles, text, edit audio, add voice overs, music and 
other special effects in a completely temporal, non-time- 
line-based composition environment. This process can 
be repeated as many times as necessary until the com- 
position is deemed satisfactory for final production in 
one or multiple formats for any number of commercial 



or personal uses. 

In contrast to conventional video and audio compo- 
sition applications, even those operating in a windowing 
environment, the primary composition user interface 
s 1 40 includes user-familiar objects, i.e. paradigms of real 
world objects that a typical user should already know 
how to use. The complexities of the operations are hid- 
den under ordinary, everyday object metaphors. All of 
the input source materials, edited materials, transitions 
10 and output are treated as objects. The objects can be 
"played" in part or in whole or caused to loop or repeat 
indefinitely at any given time during the composition 
process. In other words, the objects represent materials 
that vary over time. The primary composition user inter- 
ns face 140 provides a highly intuitive presentation of ad- 
vanced features (e.g. special effects, transitions, etc.) 
that are easily selected and invoked. 

Although a preferred embodiment of the present in- 
vention has been described, those of skill in the art will 
20 appreciate that the present invention is not limited to any 
particular application or any particular environment. The 
system and method of the present invention may be ap- 
plied advantageously to a variety of systems and appli- 
cations software, including but not limited to audio com- 
25 position, three dimensional animation, multimedia au- 
thoring and the like. Moreover, the present invention 
may be embodied on a variety of different hardware and 
software platforms, including UNIX, Macintosh™ and 
the like. 

30 Although a preferred embodiment of the present in- 
vention has been described, those of skill in the art will 
appreciate that variations and modifications may be 
made without departing from the spirit and scope thereof 
as defined by the appended claims. 
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Claims 

1. A method for non-timeline, non-linear digital multi- 
40 media composition comprising the steps of: 

a) representing multimedia materials stored in 
memory as objects: 

b) retrieving one or more of said objects from 
45 said memory for display on a composition con- 
sole; 

c) dynamically viewing and editing said one or 
more objects on said composition console to 
produce edited objects; and 

50 d) placing said edited objects on a storyline ar- 

ea of said composition console to create a sto- 
ryline, said storyline being viewable on said 
composition console. 

55 2. The method of claim 1 further comprising the steps 
of editing said storyline and producing a final com- 
position from objects on said storyline. 
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3. The method of claim 2 further comprising the step 
of storing said final composition as an object in said 
memory. 

4. The method of claim 3 wherein said step of editing 
said storyline includes steps of reorganizing objects 
thereon and dynamically viewing and editing said 
objects on said composition console. 

5. The method of claim 4 wherein said multimedia ma- 
terials includes image, video, audio, and text mate- 
rials. 

6. A system for non-timeline, non-linear digital mufti- 
media composition, the system comprising: 

a) means for representing multimedia materials 
stored in memory as objects; 

b) means for retrieving one or more of said ob- 
jects from said memory for display on a com- 
position console; 

c) means for dynamically viewing and editing 
said one or more objects on said composition 
console to produce edited objects; and 

d) means for placing said edited objects on a 
storyline area of said composition console to 
create a storyline, said storyline being viewable 
on said composition console. 

7. The system of claim 6 further comprising means for 
editing said storyline and producing a final compo- 
sition from objects on said storyline. 

8. The system of claim 7 further comprising means for 
storing said final composition as an object in said 
memory. 

9. The system of claim 8 wherein said means for ed- 
iting said storyline includes means for reorganizing 
edited objects thereon and means for dynamically 
viewing and editing said edited objects on said com- 
position console. 

10. The system of claim 9 wherein said multimedia ma- 
terials includes image, video, audio, and text mate- 
rials. 

11. A computer readable media encoded with a com- 
puter program for creating non-timeline, non-linear 
digital multimedia compositions from multimedia 
materials stored in memory as objects comprising: 

a) means for retrieving one or more of said ob- 
jects from said memory for display on a com- 
position console; 

b) means for dynamically viewing and editing 
said one or more objects on said composition 
console to produce edited objects; and 



24 

c) means for placing said edited objects on a 
storyline area of said composition console to 
create a storyline, said storyline being viewable 
on said composition console. 

12. A method for producing a multimedia composition 
comprising the steps of: 

a) providing a user interface having a source 
10 material object window and a storyline strip, 

said storyline strip having a plurality of posi- 
tions; 

b) associating individual identifiers in said win- 
dow with time varying dynamic multimedia 

15 source material objects; 

c) choosing one of said identifiers in said win- 
dow to retrieve a selected object; 

d) dragging said chosen identifier across said 
interface to one of said positions on said sto- 

20 ryline strip; 

e) repeating steps (c) and (d) to organize a mul- 
ti-object, multimedia composition in a desired 
order; and 

f) generating an output composition from se- 
25 lected objects organized on said storyline strip. 

13. The method of claim 12 wherein said positions in- 
clude object viewing frame positions and transition 
frame positions. 

30 

14. The method of claim 1 2 wherein said source mate- 
rial objects include video objects, audio objects and 
combined audio-visual objects. 

35 15. The method of claim 1 2 further comprising the step 
of moving said chosen identifier from a first position 
to a second position on said storyline strip to organ- 
ize said composition. 

40 16. The method of claim 12 wherein said output com- 
position is presented on an audio-visual display. 

17. The method of claim 12 wherein said output com- 
position is stored in external memory. 
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18. The method of claim 17 wherein said output com- 
position is stored via a tape device. 

19. The method of claim 12 wherein said source mate- 
so rial objects are retrieved from a remote source. 

20. A system for creating a multimedia composition 
from multimedia source material objects, the sys- 
tem comprising: 



55 



a) a user interface having: 

i) a tool bar for accessing system functions; 
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d) means for placing said chosen identifier at 
one of said positions on said storyline strip; 

e) means for organizing a multi-object, multi- 
media composition in a desired order; and 

& f) means for generating an output composition 

from selected objects organized on said sto- 
ryline strip. 

28. The system of claim 27 wherein said positions in- 
fo elude object viewing frame positions and transition 

frame positions. 

29. The system of claim 27 wherein said source mate- 
rial objects include video objects, audio objects and 

J 5 combined audio-visual objects. 

30. The system of claim 27 wherein said means for or- 
ganizing includes means for moving said chosen 
identifier from a first position to a second position 

20 on said storyline strip to organize said composition. 



ii) a source library window for accessing 
multimedia source material objects; 

iii) a console for previewing and editing 
said multimedia source material objects to 
create output objects; and 

iv) a storyline for assembling said output 
objects to form an output composition; 

b) a processor for editing said source material 
objects and creating and organizing said output 
composition in response to user input entered 
in a said user interface; and 

c) a display for displaying said output composi- 
tion on said console. 

21. The system of claim 20 wherein said processor is 
operable to output said output composition to an 
output device. 

22. The system of claim 20 wherein said output device 
is a memory storage medium. 

23. The system of claim 20 wherein said console in- 
cludes a preview area, and a plurality of editing tool 
activators corresponding to a plurality of editing 
tools selectable to edit said multimedia source ma- 
terial objects. 

24. The system of claim 23 wherein the activation of a 
selected one of said editing tool activators causes 
said console to display the controls of said corre- 
sponding editing tool. 

25. The system of claim 20 wherein said tool bar in- 
cludes a plurality of two dimensional iconic activa- 
tors: said iconic activators transforming into three 
dimensional activators in response to movement of 
a pointing device in close proximity. 

26. The system of claim 20 wherein said tool bar con- 
tains a plurality of two dimensional activators, said 
activators being conditioned to exhibit three dimen- 
sional rotation when activated to signal that the op- 
eration corresponding to said activator has been in- 
itialized. 



31. A computer readable media encoded with a com- 
puter program for producing a multimedia compo- 
sition from multimedia materials stored in memory 
as source material objects comprising: 

a) means for generating a user interface having 
a source material object window and a storyline 
strip, said storyline strip having a plurality of po- 
sitions; 

b) means for associating individual identifiers 
in said window with time varying dynamic mul- 
timedia source material objects; 

c) means for choosing one of said identifiers in 
said window to retrieve a selected source meth- 
od object from memory; 

d) means for placement of one or more selected 
said identifiers to selected positions on said 
storyline strip to organize a multi-object multi- 
media composition in a desired order; and 

e) means for generating an output composition 
from selected objects organized on said sto- 
ryline strip. 
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27. A system for producing a multimedia composition, 
the system comprising: 

a) means for generating a user interface having so 
a source material object window and a storyline 
strip, said storyline strip having a plurality of po- 
sitions; 

b) means for associating individual identifiers 

in said window with time varying dynamic mul- ss 
timedia source material objects; 

c) means for choosing one of said identifiers in 
said window to retrieve a selected object; 
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